A practical guide to model selection

Slides accompanying this material are available at http://clopinet.com/isabelle/ Projects/MLSS08/. We focus in the chapter on the problem of model selection. The rest of the topics covered in the class are developed in tutorials on feature selection (Guyon and Elisseeff, 2003; Guyon et al., 2006a) and causality (Guyon et al., 2007a; Guyon, 2008; Guyon et al., 2009a). Nowadays, there are many machine learning or data mining packages providing highly optimized implementations of leading algorithms, including commercial platforms like SAS and SPSS and freeware packages like Weka1, R2, Lush3 and several Matlab libraries like the Spider4 and CLOP5. The MLOSS open source repository6 indexes such valuable resources. In this context, the task of practitioners has shifted from that of identifying and implementing algorithms to that of learning how to use such packages and selecting the best model. The problem of model selection still remains largely the user’s responsibility. Best practices for model selection have emerged over the years, grounded in a wide variety of theories (regularization, Bayesian priors, MDL, structural risk minimization, bias/variance tradeoff, etc.; see Appendix A). Interestingly, all those theories converge towards the same principle stated already in the 14th century by William of Ockham: “Pluralitas non est ponenda sine neccesitate”, which prescribes limiting model complexity to the minimum necessary to explain the data, or shave off unnecessary parameters (Ockham’s razor). Indeed, as illustrated in Figure 1, attempts to improve prediction performance on training data by increasing model complexity may lead to data “over-fitting”: the good